1. Field of the Invention
The invention relates generally to images acquired with sensors (including sensors implementable on a single integrated circuit chip) responsive to luminosity information in a first spectral band (such as red, green, blue optical wavelengths, which shall be understood to include black and white) to acquire an red-green-blue (RGB) image, and responsive to wavelengths in a second spectral band, preferably near-infrared (NIR), to acquire Z data, and more particularly to improving resolution of such images.
2. Description of Related Art
Luminosity-based sensors including CMOS-implementable sensors are known in the art. Such sensors commonly include an array of pixel detectors responsive to wavelengths in a first spectral band, e.g., red, green, blue (RGB) wavelengths, which shall be understood to include black and white (RGB sensors) or simply gray scale wavelengths (black and white or BW sensors). The array can be fabricated upon an integrated circuit (IC) substrate upon which may be fabricated analog-to-digital conversion circuitry and signal processing circuitry. While such luminosity based sensors can provide a color (RGB) or gray scale (BW) image, they provide no useful depth information.
FIG. 1A depicts an exemplary application of a conventional RGB or BW sensor. Suppose that it is desired to use a camera system to intelligently recognize objects within a field of view. In some applications the camera system might be provided in or on a motor vehicle to scan the road ahead for target objects that might be endangered by the motor vehicle, pedestrians perhaps. An exemplary camera system 5 includes a lens 10 that receives red, green, and blue components of visible light energy 15 reflected from a target object 20 a distance Z away from the camera system. Associated with the camera system is a prior art RGB sensor array 25 that outputs a signal responsive to the incoming RGB light components. Resolution of RGB sensor array 25 is a function of sensor area. A typical sensor array 25 may comprise more than a million pixels, each pixel being fairly small in area, perhaps less than 5 μm·5 μm. RGB sensors of at least 1 MP (megapixel) area are relatively inexpensive to mass produce because of the small size of the individual pixels. In FIG. 1A, sensor circuit 30 receives an output signal from sensor 25 and attempts to identify the target object 20 in terms of hazard potential. If desired, output from the camera system may include an image 35 electronically generated, for example on a flat screen monitor screen (or in some applications printed on media such as paper). In a hazard warning application, image 35 might be displayed within a red circle to designate an immediate hazard to be avoided, and sensor circuit 60 may also cause audible warnings to be sounded.
Although resolution of RGB sensor 25 may be adequate to display target object 20, rapid identification of the nature and size of the target would be improved if Z data, usually acquired from IR wavelengths, could also be used. Such information, if available, could also be used to provide a measure of the actual size of the target object.
It is also known in the art to fabricate range-finding or three-dimensional sensors using an array of pixel detectors, e.g., U.S. Pat. No. 6,323,942 (2001) entitled CMOS-Compatible Three-Dimensional Image Sensor IC, U.S. Pat. No. 6,515,740 (2003) entitled Methods for CMOS-Compatible Three-Dimensional Imaging Sensing Using Quantum Efficiency Modulation (2003) and U.S. Pat. No. 6,580,496 (2003) entitled Systems for CMOS-Compatible Three-Dimensional Imaging Sensing Using Quantum Efficiency Modulation. These three patents are assigned to assignee herein Canesta, Inc., now of Sunnyvale, Calif. These patents disclose sensor systems that provide depth information (Z-distance between the sensor and a target object) at each pixel detector in the sensor array for each frame of acquired data. Z-range-finding detectors according to the '942 patent determine range z by measuring time-of-flight (TOF) between emission of pulsed optical energy and detection of target object reflected optical energy. Z-range-finding systems according to the '740 and '496 patents operate somewhat similarly but detect phase shift between emitted and reflected-detected optical energy to determine z-range. Detection of the reflected optical energy signals over multiple locations in the pixel array results in measurement signals that are referred to as depth images.
FIG. 1B is taken from the '740 and '496 patents and depicts a phase-shift TOF system 40 fabricated on an IC 45 that includes a two-dimensional array 55 of pixel detectors 60, each of which has dedicated circuitry 65 for processing detection charge output by the associated detector. Array 55 includes a number r of rows and a number c of column of pixels, where the array size, e.g., 32×32, 64×564, 140×160, etc., defines the x-y plane resolution of the sensor. Unfortunately it is difficult to readily implement a large, high resolution array 55 because of the relatively large area required for the individual pixels, perhaps 50 μm·50 μm. (Recall that the relatively small 5 μm·5 μm size for RGB pixels readily allowed implementing arrays of greater than 1 MP.) IC 45 also includes a microprocessor or microcontroller unit 70, memory 72 (which preferably includes random access memory or RAM and read-only memory or ROM), a high speed distributable clock 74, and various computing and input/output (I/O) circuitry 76. Among other functions, controller unit 70 may perform distance to object and object velocity calculations.
Under control of microprocessor 70, a source of optical energy 50 is periodically energized by an exciter 80, and emits optical energy via lens 85 toward an object target 20. Emitter 50 preferably is an LED or laser diode emitting low power (e.g., perhaps 50 mW peak) periodic waveform, producing optical energy emissions of known frequency (perhaps a few hundred MHz) for a time period known as the shutter time (perhaps 10 ms). Typically emitter 50 operates at IR, which is understood to include near IR, e.g., perhaps 800 nm).
Some of the emitted optical energy (denoted S1) will be reflected off the surface of target object 20 (denoted S2), will pass through an aperture field stop and lens, collectively 90, and falls upon two-dimensional array 55 of pixel detectors 60 to form a depth image.
For each pulse or burst of optical energy transmitted by emitter 50, a three-dimensional image of the visible portion of target object 20 is acquired, from which intensity (A) and Z data is obtained (DATA). More specifically, reflected incoming optical energy S2 detected by each imaging pixel detectors 60 includes intensity information (A), and phase shift information (Φ), where phase shift Φ varies with distance Z and can be processed to yield Z data. The time-of-flight (TOF) required for optical energy transmitted by emitter 50 to reach target object 20 and be reflected back and detected by pixel detectors 60 is denoted as t. TOF information is captured from which distances Z are determined from the relationship Z1=t·C/2, where Z is distance to be measured, t is roundtrip TOF time, and C is velocity of light. TOF sensor system 40 can acquire three-dimensional images of a target object in real time, simultaneously acquiring both luminosity data (e.g., signal amplitude) and true TOF distance measurements of a target object or scene. In FIG. 1B, Energy from emitter 50 and detected signals within pixel detectors 60 are synchronous to each other such that phase difference and thus distance Z can be measured for each pixel detector. If the phase-detection aspect of FIG. 1A were omitted, the resultant system would be essentially that of U.S. Pat. No. 6,323,942 in which pulses or bursts of optical energy are emitted and target-object reflected portions of that energy are detected and TOF is literally counted.
In many applications it can be important to simultaneously acquire from a single field of view or bore sight both data in a first spectral band, typically RGB data (used to provide an RGB image) and Z data (preferably acquired at in a second spectral band, typically IR wavelengths). But this goal is difficult to attain in practice because, as noted above, pixel detectors used to capture Z-data at IR wavelengths are commonly much larger in area than pixel detectors responsive to RGB wavelengths, an area that is perhaps 100 times larger. If a single detector array were fabricated to simultaneously use RGB pixel detectors and Z pixel detectors, the presence of the large sized Z pixel detectors in a high density array of much smaller sized RGB pixel detectors would cause large image artifacts that could degrade the quality of a resultant RGB image. Further, pixel detectors responsive to Z data often require high quality (preferably IR wavelength) bandpass filtering. In practice, CMOS fabrication does not presently implement such bandpass filtering for the Z pixels, especially with desired narrow band characteristics that may be on the order of 50 nm or less.
Applicants's co-pending application Ser. No. 11/044,996 was directed to a CMOS-implementable sensor that included pixel detectors responsive to wavelengths in a first spectral band, such as RGB wavelengths, and that also included pixel detectors responsive to preferably Z data in a second spectral band, preferably NIR wavelengths. Preferably such sensor array should be implementable on a single IC substrate.
Relevant to the present invention, in some applications there is a need to provide a low cost z-sensing system (or camera) that includes a first z-sensor array whose normally low x-y resolution can be increased by combining this first sensor array with a second lower cost, higher resolution RGB sensor array (or camera).
While such increased x-y resolution could be achieved by fabricating a single sensor array combining large area z-sensors with smaller area RGB sensors, preferably increased x-y resolution should also be achieved using a z-sensor array and a separate RGB sensor array, or sensor arrays as described in U.S. patent application Ser. No. 11/044,996. Further, such increased x-y resolution should be achieved without undue constraints with regard to the physical disposition of the first and second sensor arrays, e.g., without regard to whether the two arrays are disposed in mono or in stereoscopic relationship to each other.
The present invention provides such enhanced x-y resolution in a cost effective fashion, by substantially simultaneously using a fusion algorithm to combine output from a first, low resolution, z-sensor array, with output from a second, higher resolution, inexpensive RGB sensor array to yield a single frame of high resolution z-data, which high resolution may be greater than or equal to the RGB sensor resolution.